From Perceptual Relations to Scene Gist Recognition

The ability to recognize visual scenes quickly and accurately is highly constructive for both biological and machine vision. In this work we study the process of scene gist recognition from a novel point of view and investigate whether prior knowledge of the perceptual relations between the different scene categories may help facilitate better computational models for scene gist recognition. We first introduce a psychophysical paradigm that probes human scene gist recognition and extracts perceptual relations between scene categories. Then, we show that these perceptual relations do not always conform the semantic structure between categories. Next, we incorporate the obtained perceptual relations into a computational classification scheme, which takes inter-class relationships into account to obtain better scene recognition regardless of the particular descriptors with which scenes are represented. We present such improved recognition performance using several popular descriptors, we discuss why the contribution of inter-class perceptual relations is particularly pronounced for under-sampled training sets, and we argue that this mechanism may explain the ability of the human visual system to perform well under similar conditions. Finally, we introduce an online experimental system for obtaining perceptual relations for large collections of scene categories. 1. Perceptual Relations A key issue in the context of scene gist recognition is perceptual relations (as opposed to semantic relations; see below), a possibility that has been rarely considered either in the perceptual literature or the computational literature [7, 2, 8]. However, even intuitively, when our visual system observes a bedroom scene for a fraction of a second and “deliberates” how to categorize it, what possibly comes to mind in addition to “bedroom” are perhaps classes like “living room” or “kitchen”. It appears as if our visual system does not even consider possibilities such as “coast” or “highway”, or more generally, scenes which are perceptually “distant” from the observable reference class. Put differently, prior knowledge about the perceptual relations between the different categories of scenes may help facilitate more accurate and more efficient scene gist recognition. Knowledge of such relationships could also partly explain the fact that humans are often able to learn and process hundreds of scene categories from very few training examples while computational models usually need at least tens of training examples per category before achieving reasonable recognition performance. Exploring relations between categories is not new and was recently promoted by exploiting WordNet as a semantic relationships database for object recognition [3]. Indeed, semantic relationships can be extracted quite conveniently from WordNet. Still, we found several examples to suggest that semantic relationships between categories do not necessarily agree with their perceptual relationships. For example, our experimental setup reveals that the “highway” category is perceptually closer to “coast” than to “kitchen”, although semantically the opposite holds. Once relations between visual categories are considered based on perceptual criteria, two questions immediately arise. First, how can perceptual difference or distance between scene categories be determined or inferred directly from human vision? Put differently, can the perceptual distance between categories be measured psychophysically in a robust and unbiased way? Second, once determined, how could these perceptual relations be incorporated into a computational classification scheme. We introduce a psychophysical paradigm where we briefly present two natural scene stimuli simultaneously and ask human observers whether they belong to the same scene category or not (i.e., same/different forced choice task). Collected from 79 human subjects, we analyze subjects’ average performance over to provide an unbiased objective measure regarding the perceptual “distance” between the different scene categories. In particular, we calculate subjects’ probability to respond Different for each pair of categories. Since this probability is expected to increase when such judgment is easier, and since the latter case is expected when scenes become more “perceptually different”, this probability is termed as the “perceptual distance” (PD)